A service provider may lease to a client portions of computing resources that the service provider manages in a service provider network. The service provider manages the operation of the computing resources to allow the resources to be efficiently and securely shared by multiple customers. As part of its management, the service provider may collect and track usage and performance metrics generated from the computer resources. In addition, the service provider may use prediction models to make predictions about the metrics at particular times in the future. For example, a prediction model may be used to predict the storage utilization for a storage resource at a time in the future. Such information may be used by a client of the storage resource to gradually scale the storage resource. The service provider may allow clients to use its prediction models through a prediction service offered by the service provider.
The service provider may operate the prediction service based on a prediction model in a production environment. From time to time, the service provider may modify the prediction service by updating the prediction model or replacing it with a new prediction model. However, it is generally difficult to determine how a new prediction model will perform in an actual production environment using real world metrics data. For one thing, an evaluation of prediction model performance is not simply a matter of correctness. Moreover, it may be desirable in some systems that a new prediction model does not generate predictions that are drastically different from the current prediction model in the production environment. Further, a new model may need to be observed over a long period of time under realistic operating conditions. For this reason, the performance evaluation is not a step that is easily incorporated into a traditional software testing platform. These issues cause difficulties in the testing of the prediction service, adding to both the time and cost needed for the testing process.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include,” “including,” and “includes” mean including, but not limited to.
It will also be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the present invention. The first contact and the second contact are both contacts, but they are not the same contact.